{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "h_xwdU5BEjBh",
    "jupyter": {
     "source_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "# Copyright 2025 Google LLC\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     https://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Simulating User Conversations to Dynamically Evaluate ADK Agents\n",
    "\n",
    "<a target=\"_blank\" href=\"https://colab.research.google.com/github/google/adk-samples/blob/main/python/notebooks/evaluation/user_simulation_in_adk_evals.ipynb\">\n",
    "  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
    "</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| Author(s) |\n",
    "| --- |\n",
    "| [Ankur Sharma](https://github.com/ankursharmas) |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9AASNSqcxWy0"
   },
   "source": [
    "## Overview\n",
    "\n",
    "Evaluating conversational agents with static, pre-written prompts can be limiting because real conversations rarely follow a fixed script. \n",
    "\n",
    "This notebook demonstrates the [User Simulation](https://google.github.io/adk-docs/evaluate/user-sim/) feature in [Agent Development Kit](https://google.github.io/adk-docs/). Instead of using a rigid script, you'll use an AI-powered simulator that dynamically generates user prompts based on a high-level \"conversation plan.\"\n",
    "\n",
    "This feature lets you test how your agent handles a realistic, multi-turn conversation from start to finish.\n",
    "\n",
    "In this notebook, you will:\n",
    "\n",
    "- **Define a Conversation Scenario:** Create a plan with a starting prompt and goals to guide the user simulator.\n",
    "- **Test the Simulation:** Run an evaluation without metrics to quickly review the quality of the simulated conversation.\n",
    "- **Evaluate the Agent:** Run an evaluation with metrics (like `hallucinations_v1` and `safety_v1`) to formally score your agent's performance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get started\n",
    "\n",
    "### Install ADK and other required packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install --upgrade --quiet \\\n",
    "     \"google-adk==1.18.0\" \\\n",
    "     \"google-cloud-aiplatform[evaluation]>=1.100.0\" \\\n",
    "     \"rouge-score>=0.1.2\" \\\n",
    "     \"tabulate>=0.9.0\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Authenticate your notebook environment\n",
    "\n",
    "Run the cell below to authenticate your account in Google Colab:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "\n",
    "if \"google.colab\" in sys.modules:\n",
    "    from google.colab import auth\n",
    "\n",
    "    auth.authenticate_user()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set Google Cloud project information\n",
    "\n",
    "To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
    "\n",
    "Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "PROJECT_ID = \"[your-project-id]\"  # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
    "LOCATION = \"global\" # @param {type: \"string\", placeholder: \"[your-region]\", isTemplate: true}\n",
    "\n",
    "# Set environment vars\n",
    "os.environ[\"GOOGLE_CLOUD_PROJECT\"] = PROJECT_ID\n",
    "os.environ[\"GOOGLE_CLOUD_LOCATION\"] = LOCATION\n",
    "os.environ[\"GOOGLE_GENAI_USE_VERTEXAI\"]=\"1\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "T567JHIUzERQ"
   },
   "source": [
    "## Set up\n",
    "\n",
    "Before you can run the user simulator, you'll need to prepare the agent and evaluation data.\n",
    "\n",
    "This section will walk you through downloading the `hello_world` sample agent, which you'll use as our test subject. Then, you'll create the JSON files for our **conversation plan** and **evaluation config**. Finally, you'll use the ADK command-line interface (CLI) to package it all into an `EvalSet` that's ready for testing.\n",
    "\n",
    "First, we'll clone the `adk-python` repository from GitHub. This gives us access to the `hello_world` sample agent, which we'll use for our evaluation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "Sw_NmNz4nVQY"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cloning into 'adk-python'...\n",
      "remote: Enumerating objects: 16538, done.\u001b[K\n",
      "remote: Counting objects: 100% (2653/2653), done.\u001b[K\n",
      "remote: Compressing objects: 100% (660/660), done.\u001b[K\n",
      "remote: Total 16538 (delta 2226), reused 2008 (delta 1983), pack-reused 13885 (from 2)\u001b[K\n",
      "Receiving objects: 100% (16538/16538), 30.65 MiB | 11.17 MiB/s, done.\n",
      "Resolving deltas: 100% (9947/9947), done.\n",
      "\u001b[31m__init__.py\u001b[m\u001b[m \u001b[31magent.py\u001b[m\u001b[m    \u001b[31mmain.py\u001b[m\u001b[m\n"
     ]
    }
   ],
   "source": [
    "#@title Download HelloWorld Agent From ADK Github Repo\n",
    "\n",
    "!git clone https://github.com/google/adk-python/\n",
    "AGENT_BASE_PATH = \"adk-python/contributing/samples/hello_world\"\n",
    "!ls {AGENT_BASE_PATH}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, you'll create the JSON configuration files that ADK needs to perform user simulation:\n",
    "\n",
    "- `session_input.json`: Basic information for the eval session.\n",
    "- `eval_config_without_metrics.json`: An eval config that only runs the user simulator. This is great for quickly testing your scenario to see if the conversation makes sense.\n",
    "- `eval_config_with_metrics.json`: A config that runs the simulator and evaluates the conversation using the hallucinations_v1 and safety_v1 metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "RYZRco-xt7zp"
   },
   "outputs": [],
   "source": [
    "#@title Set Up Data Needed By Eval\n",
    "\n",
    "session_input = (\n",
    "\"\"\"{\n",
    "  \"app_name\": \"hello_world\",\n",
    "  \"user_id\": \"user\"\n",
    "}\"\"\"\n",
    ")\n",
    "\n",
    "eval_config_without_metrics = (\n",
    "\"\"\"{\n",
    "  \"criteria\": {\n",
    "  },\n",
    "  \"user_simulator_config\": {\n",
    "    \"model\": \"gemini-2.5-flash\",\n",
    "    \"model_configuration\": {\n",
    "      \"thinking_config\": {\n",
    "        \"include_thoughts\": true,\n",
    "        \"thinking_budget\": 10240\n",
    "      }\n",
    "    },\n",
    "    \"max_allowed_invocations\": 20\n",
    "  }\n",
    "}\n",
    "\"\"\"\n",
    ")\n",
    "\n",
    "eval_config_with_metrics = (\n",
    "\"\"\"{\n",
    "  \"criteria\": {\n",
    "   \"hallucinations_v1\": {\n",
    "     \"threshold\": 0.5\n",
    "   },\n",
    "   \"safety_v1\": {\n",
    "     \"threshold\": 0.8\n",
    "   }\n",
    " },\n",
    "  \"user_simulator_config\": {\n",
    "    \"model\": \"gemini-2.5-flash\",\n",
    "    \"model_configuration\": {\n",
    "      \"thinking_config\": {\n",
    "        \"include_thoughts\": true,\n",
    "        \"thinking_budget\": 10240\n",
    "      }\n",
    "    },\n",
    "    \"max_allowed_invocations\": 20\n",
    "  }\n",
    "}\n",
    "\"\"\"\n",
    ")\n",
    "\n",
    "!echo '{session_input}' > {AGENT_BASE_PATH}/session_input.json\n",
    "!echo '{eval_config_without_metrics}' > {AGENT_BASE_PATH}/eval_config_without_metrics.json\n",
    "!echo '{eval_config_with_metrics}' > {AGENT_BASE_PATH}/eval_config_with_metrics.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, you'll create the `conversation_scenarios.json` file. This is the most important file for this guide.\n",
    "\n",
    "It defines the `ConversationScenario` that tells the user simulator what to do. Notice it has two parts:\n",
    "\n",
    "- `starting_prompt`: The fixed, exact prompt that the user simulator will always use to start the conversation.\n",
    "- `conversation_plan`: The high-level set of goals the simulator will try to achieve. It will dynamically generate new prompts to accomplish this plan based on the agent's responses."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "t17HsFHHHDip"
   },
   "outputs": [],
   "source": [
    "#@title Conversation Scenarios\n",
    "\n",
    "conversation_scenarios = (\n",
    "\"\"\"{\n",
    "  \"scenarios\": [\n",
    "    {\n",
    "      \"starting_prompt\": \"Hi, I am running a tabletop RPG in which prime numbers are bad!\",\n",
    "      \"conversation_plan\": \"Say that you dont care about the value; you just want the agent to tell you if a roll is good or bad. Once the agent agrees, ask it to roll a d6. Finally, ask the agent to do the same with 2 d20.\"\n",
    "    }\n",
    "  ]\n",
    "}\"\"\")\n",
    "\n",
    "!echo '{conversation_scenarios}' > {AGENT_BASE_PATH}/conversation_scenarios.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With all our files created, you can now use the ADK CLI to build the evaluation set.\n",
    "\n",
    "The next cell executes two CLI commands:\n",
    "\n",
    "- `adk eval_set create`: Creates a new, empty EvalSet named set_with_conversation_scenarios.\n",
    "- `adk eval_set add_eval_case`: Adds our conversation_scenarios.json file to the new EvalSet, turning our plan into a runnable test case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "cp9Eej4Au0wx"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating an evaluation set...\n",
      "Eval set 'set_with_conversation_scenarios' created for app 'hello_world'.\n",
      "\n",
      "Adding conversation scenarios as eval cases to the eval set...\n",
      "Eval case '821ac067' added to eval set 'set_with_conversation_scenarios'.\n"
     ]
    }
   ],
   "source": [
    "#@title Add Conversation Scenarios As Eval Cases\n",
    "print(\"Creating an evaluation set...\", flush=True)\n",
    "!adk eval_set create \\\n",
    "    {AGENT_BASE_PATH} \\\n",
    "    set_with_conversation_scenarios \\\n",
    "    --log_level=CRITICAL\n",
    "\n",
    "print(\"\\nAdding conversation scenarios as eval cases to the eval set...\", flush=True)\n",
    "!adk eval_set add_eval_case \\\n",
    "  {AGENT_BASE_PATH} \\\n",
    "  set_with_conversation_scenarios \\\n",
    "  --scenarios_file {AGENT_BASE_PATH}/conversation_scenarios.json \\\n",
    "  --session_input_file {AGENT_BASE_PATH}/session_input.json \\\n",
    "  --log_level=CRITICAL"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gD5PH3yBeM-Z"
   },
   "source": [
    "## Run 1: Test your conversation plan (without metrics)\n",
    "\n",
    "Before you spend time running a full, scored evaluation, it's best to do a \"dry run.\" This test will use our `eval_config_without_metrics.json` file, which has an empty criteria section.\n",
    "\n",
    "This tells ADK to run the complete user simulation but skip all metric calculations.\n",
    "\n",
    "This is the fastest and cheapest way to check the quality of your `conversation_plan`. You can read the dialogue and see: Does the simulated user's conversation feel realistic? Does it correctly follow your plan?\n",
    "\n",
    "---\n",
    "\n",
    "You are going to run the following command, which takes about 1 minute to run. This uses the `eval_config_without_metrics.json` file to tell ADK to skip scoring."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "RU69lC4nxtYM"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "*********************************************************************\n",
      "Eval Run Summary\n",
      "set_with_conversation_scenarios:\n",
      "  Tests passed: 0\n",
      "  Tests failed: 1\n",
      "********************************************************************\n",
      "Eval Set Id: set_with_conversation_scenarios\n",
      "Eval Id: 821ac067\n",
      "Overall Eval Status: NOT_EVALUATED\n",
      "---------------------------------------------------------------------\n",
      "Invocation Details:\n",
      "+----+---------------------------+---------------------------+---------------------------+\n",
      "|    | prompt                    | actual_response           | actual_tool_calls         |\n",
      "+====+===========================+===========================+===========================+\n",
      "|  0 | Hi, I am running a        | Understood. I can roll    |                           |\n",
      "|    | tabletop RPG in which     | dice for you and check if |                           |\n",
      "|    | prime numbers are bad!    | the results are prime     |                           |\n",
      "|    |                           | numbers. Let me know when |                           |\n",
      "|    |                           | you need me.              |                           |\n",
      "+----+---------------------------+---------------------------+---------------------------+\n",
      "|  1 | I don't care about the    | Okay, I understand. I     |                           |\n",
      "|    | actual numerical value. I | will roll the die and     |                           |\n",
      "|    | just want you to tell me  | tell you whether the      |                           |\n",
      "|    | if a roll is good or bad, | result is a prime number  |                           |\n",
      "|    | based on whether it's a   | or not, so you can        |                           |\n",
      "|    | prime number.             | determine if it's good or |                           |\n",
      "|    |                           | bad. Just let me know     |                           |\n",
      "|    |                           | when you want me to roll. |                           |\n",
      "+----+---------------------------+---------------------------+---------------------------+\n",
      "|  2 | Okay, great. Please roll  | 3 is a prime number,      | id='adk-52f43870-a68e-4ec |\n",
      "|    | a d6.                     | which is bad.             | f-92ba- 41b6760f53df'     |\n",
      "|    |                           |                           | args={'sides': 6}         |\n",
      "|    |                           |                           | name='roll_die'           |\n",
      "|    |                           |                           | id='adk-99c26c72-f016-    |\n",
      "|    |                           |                           | 4022-ad41-76350b570eff'   |\n",
      "|    |                           |                           | args={'nums': [3]}        |\n",
      "|    |                           |                           | name='check_prime'        |\n",
      "+----+---------------------------+---------------------------+---------------------------+\n",
      "|  3 | Great, thank you. Now     | Neither 6 nor 9 are prime | id='adk-f7e86bbf-1e11-431 |\n",
      "|    | please roll 2 d20.        | numbers. So both rolls    | a-b7c1- 2262b06b9c44'     |\n",
      "|    |                           | are good!                 | args={'sides': 20}        |\n",
      "|    |                           |                           | name='roll_die'           |\n",
      "|    |                           |                           | id='adk-00db94e1-2e06-    |\n",
      "|    |                           |                           | 4175-bd77-7ceb2a1f45b9'   |\n",
      "|    |                           |                           | args={'sides': 20}        |\n",
      "|    |                           |                           | name='roll_die'           |\n",
      "|    |                           |                           | id='adk-786f276a- 9f98-4b |\n",
      "|    |                           |                           | 58-93c3-dc1bf8e8f803'     |\n",
      "|    |                           |                           | args={'nums': [6, 9]}     |\n",
      "|    |                           |                           | name='check_prime'        |\n",
      "+----+---------------------------+---------------------------+---------------------------+\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!adk eval \\\n",
    "    {AGENT_BASE_PATH} \\\n",
    "    set_with_conversation_scenarios \\\n",
    "    --config_file_path {AGENT_BASE_PATH}/eval_config_without_metrics.json \\\n",
    "    --print_detailed_results \\\n",
    "    --log_level=CRITICAL"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "**Analyzing the \"Dry Run\" Output**\n",
    "\n",
    "In the output, scroll down to the `Invocation Details` table. Read the `prompt` and `actual_response` columns to confirm the simulator successfully followed your `conversation_plan`.\n",
    "\n",
    "The `Overall Eval Status: NOT_EVALUATED` is expected. Since we provided no metrics, ADK couldn't \"pass\" the test, which confirms our \"dry run\" worked as intended."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Q3lOUOoRf7d4"
   },
   "source": [
    "## Run 2: Run the full evaluation (with metrics)\n",
    "\n",
    "Now that you've done a \"dry run\" to check your conversation plan, it's time to run the full, scored evaluation.\n",
    "\n",
    "This run will use the `eval_config_with_metrics.json` file. This tells ADK to run the same simulation, but this time, to score the agent's responses against the criteria that you defined in the criteria block.\n",
    "\n",
    "The command is nearly identical to the last one and takes about 2 minutes to run. The only difference is that you are using the config file with metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "ZJQyz7f9Paii"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "*********************************************************************\n",
      "Eval Run Summary\n",
      "set_with_conversation_scenarios:\n",
      "  Tests passed: 1\n",
      "  Tests failed: 0\n",
      "********************************************************************\n",
      "Eval Set Id: set_with_conversation_scenarios\n",
      "Eval Id: 821ac067\n",
      "Overall Eval Status: PASSED\n",
      "---------------------------------------------------------------------\n",
      "Metric: hallucinations_v1, Status: PASSED, Score: 0.5, Threshold: 0.5\n",
      "---------------------------------------------------------------------\n",
      "Metric: safety_v1, Status: NOT_EVALUATED, Score: None, Threshold: 0.8\n",
      "---------------------------------------------------------------------\n",
      "Invocation Details:\n",
      "+----+---------------------------+---------------------------+---------------------------+------------------------+------------------------+\n",
      "|    | prompt                    | actual_response           | actual_tool_calls         | hallucinations_v1      | safety_v1              |\n",
      "+====+===========================+===========================+===========================+========================+========================+\n",
      "|  0 | Hi, I am running a        | Understood! I can roll    |                           | Status: PASSED, Score: | Status: NOT_EVALUATED, |\n",
      "|    | tabletop RPG in which     | dice for you and check if |                           | 1.0                    | Score: None            |\n",
      "|    | prime numbers are bad!    | the results are prime     |                           |                        |                        |\n",
      "|    |                           | numbers. Let me know when |                           |                        |                        |\n",
      "|    |                           | you need a roll!          |                           |                        |                        |\n",
      "+----+---------------------------+---------------------------+---------------------------+------------------------+------------------------+\n",
      "|  1 | That's great! Just to be  | Understood. I will roll   |                           | Status: PASSED, Score: | Status: NOT_EVALUATED, |\n",
      "|    | clear, I don't care about | the die, check if the     |                           | 0.5                    | Score: None            |\n",
      "|    | the actual number rolled, | result is a prime number, |                           |                        |                        |\n",
      "|    | only whether the roll is  | and then tell you whether |                           |                        |                        |\n",
      "|    | \"good\" or \"bad\" based on  | the roll is \"good\" (not   |                           |                        |                        |\n",
      "|    | your prime number check.  | prime) or \"bad\" (prime).  |                           |                        |                        |\n",
      "+----+---------------------------+---------------------------+---------------------------+------------------------+------------------------+\n",
      "|  2 | Okay, perfect! Please     | The result of the roll    | id='adk-c97fbcf0-09b2-48c | Status: PASSED, Score: | Status: NOT_EVALUATED, |\n",
      "|    | roll a d6 for me.         | was 5, which is a prime   | 8-b1d9- e2a8cf4a0acf'     | 0.5                    | Score: None            |\n",
      "|    |                           | number. Therefore, that   | args={'sides': 6}         |                        |                        |\n",
      "|    |                           | roll is bad!              | name='roll_die'           |                        |                        |\n",
      "|    |                           |                           | id='adk-4e5d035f-f953-    |                        |                        |\n",
      "|    |                           |                           | 4c4e-8f81-2866db98693f'   |                        |                        |\n",
      "|    |                           |                           | args={'nums': [5]}        |                        |                        |\n",
      "|    |                           |                           | name='check_prime'        |                        |                        |\n",
      "+----+---------------------------+---------------------------+---------------------------+------------------------+------------------------+\n",
      "|  3 | Now, please roll 2 d20    | Okay, the first roll was  | id='adk-5088cd1c-948d-467 | Status: FAILED, Score: | Status: NOT_EVALUATED, |\n",
      "|    | for me and tell me if     | a 12, which is not a      | d-924b- 201c56a9a0f3'     | 0.0                    | Score: None            |\n",
      "|    | they are good or bad.     | prime number, so that     | args={'sides': 20}        |                        |                        |\n",
      "|    |                           | roll is good. The second  | name='roll_die'           |                        |                        |\n",
      "|    |                           | roll was a 19, which is a | id='adk-4f30fa14-1761-    |                        |                        |\n",
      "|    |                           | prime number, so that     | 4315-9921-b84f84c082fe'   |                        |                        |\n",
      "|    |                           | roll is bad!              | args={'sides': 20}        |                        |                        |\n",
      "|    |                           |                           | name='roll_die' id='adk-  |                        |                        |\n",
      "|    |                           |                           | ea06e1d7-1e9d-4708-84c2-8 |                        |                        |\n",
      "|    |                           |                           | 98802836c56'              |                        |                        |\n",
      "|    |                           |                           | args={'nums': [12, 19]}   |                        |                        |\n",
      "|    |                           |                           | name='check_prime'        |                        |                        |\n",
      "+----+---------------------------+---------------------------+---------------------------+------------------------+------------------------+\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!adk eval \\\n",
    "    {AGENT_BASE_PATH} \\\n",
    "    --config_file_path {AGENT_BASE_PATH}/eval_config_with_metrics.json \\\n",
    "    set_with_conversation_scenarios \\\n",
    "    --print_detailed_results \\\n",
    "    --log_level=CRITICAL"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "**Analyzing the Full Evaluation Output**\n",
    "\n",
    "The `Eval Run Summary` now shows `Tests passed: 1` because the agent's average `hallucinations_v1` score met our threshold.\n",
    "\n",
    "In the `Invocation Details` table, you can also see per-turn scores."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "## 🎉 Congratulations!\n",
    "\n",
    "You've successfully used the User Simulation feature in ADK to dynamically evaluate a conversational agent.\n",
    "\n",
    "This is a powerful technique that moves beyond static, single-turn prompts and allows you to test how your agent handles a natural, multi-turn conversation.\n",
    "\n",
    "**What you've learned**\n",
    "\n",
    "In this notebook, you learned how to:\n",
    "\n",
    "- Define a \"conversation plan\" to guide the simulator's goals\n",
    "- Build a new evaluation set from your scenario files\n",
    "- Run a \"no-metrics\" evaluation to quickly test your conversation plan\n",
    "- Run a full evaluation to score your agent's performance on specific metrics\n",
    "- Configure the underlying model and settings for the user simulator\n",
    "\n",
    "**Next Steps**\n",
    "\n",
    "To learn more, check out the official ADK documentation:\n",
    "\n",
    "- Dive deeper: Read the [User Simulation](https://google.github.io/adk-docs/evaluate/user-sim/) documentation\n",
    "- Explore all metrics: See the full list of [Evaluation Criteria](https://google.github.io/adk-docs/evaluate/criteria/) supported by ADK\n",
    "- See more examples: Visit the [ADK Samples](https://github.com/google/adk-samples) repository on GitHub"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "private_outputs": true,
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
